Method and System for Power-Aware Motion Estimation for Video Processing

ABSTRACT

Methods and systems for power-aware motion estimation video processing are disclosed. Aspects of one method may include estimating motion for video data by block matching reduced resolution blocks of video data to generate an initial motion vector. The preliminary motion vector and motion for a previous frame may be used to generate a final motion vector for the block for the present frame using an iterative algorithm. The motion estimation may be dynamically enabled and/or disabled based on content of the video data, available power to a mobile terminal, and/or a user input. The iterations used to generate the final motion vector may be based on content of the video data, available power to a mobile terminal, and/or a user input.

CROSS-REFERENCE TO RELATED APPLICATIONS/INCORPORATION BY REFERENCE

This application is a continuation of U.S. patent application Ser. No.11/871,370, entitled “METHOD AND SYSTEM FOR POWER-AWARE MOTIONESTIMATION FOR VIDEO PROCESSING,” and filed on Oct. 12, 2007, which isincorporated by reference in its entirety.

FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[Not Applicable]

MICROFICHE/COPYRIGHT REFERENCE

[Not Applicable]

FIELD OF THE INVENTION

Certain embodiments of the invention relate to video processing. Morespecifically, certain embodiments of the invention relate to a methodand system for power-aware motion estimation video processing.

BACKGROUND OF THE INVENTION

Users of electronic devices are becoming more aware of power consumptionby those devices. Certainly, users of mobile devices are sensitive tothe issue of power consumption since the mobile devices are inherentlylimited in available battery power, and the battery needs to berecharged periodically. Additionally, the “green” movement is leading toreduced power usage even for electronic devices that are plugged in.Reduced power requirement will benefit the consumer with reducedelectric bills, and benefit society with reduced power demand on apower-grid. However, generally, when an electronic device enters alow-power state, functionality of the electronic device is greatlyreduced.

Further limitations and disadvantages of conventional and traditionalapproaches will become apparent to one of skill in the art, throughcomparison of such systems with some aspects of the present invention asset forth in the remainder of the present application with reference tothe drawings.

BRIEF SUMMARY OF THE INVENTION

A system and/or method for power-aware motion estimation videoprocessing, substantially as shown in and/or described in connectionwith at least one of the figures, as set forth more completely in theclaims.

Various advantages, aspects and novel features of the present invention,as well as details of an illustrated embodiment thereof, will be morefully understood from the following description and drawings.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is an exemplary block diagram of a portion of a mobile terminal,in accordance with an embodiment of the invention.

FIG. 2A is an exemplary image lattice illustrating initial motion vectorlocations, in accordance with an embodiment of the invention.

FIG. 2B is an exemplary image lattice illustrating determining motionvector locations, in accordance with an embodiment of the invention.

FIG. 3 is an exemplary block diagram illustrating data flow forpower-aware motion estimation, in accordance with an embodiment of theinvention.

FIG. 4 is an exemplary graph illustrating power consumption versusnumber of iterations in determining motion estimation, in accordancewith an embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

Certain embodiments of the invention may be found in a method and systemfor power-aware motion estimation video processing. Aspects of theinvention may comprise estimating motion for video data by blockmatching a reduced resolution block of video data in a present framewith one or more similarly reduced resolution blocks of video data in aprevious frame to generate a preliminary motion vector for the block ofvideo data in the present frame. The preliminary motion vector andmotion vectors that correspond to blocks of video data from a previousframe may be used to generate a final motion vector for the block ofvideo data for the present frame. A global motion vector may also beused in generating the final motion vector. The final motion vector maybe generated using an iterative algorithm that adds the preliminarymotion vector to a feedback motion vector. The feedback motion vectormay be generated by appropriately processing the result of adding thepreliminary motion vector to a feedback motion vector. The final motionvector for each block of video data may be stored in memory forgenerating motion vectors for blocks of video data in a subsequentframe.

The motion estimation may be dynamically enabled and/or disabled basedon content of the video data, available power to a mobile terminal,and/or a user input. The number of iterations used for the iterativealgorithm in generating the final motion vector may also be based oncontent of the video data, available power to a mobile terminal, and/ora user input. The incremental correction for an output generated at eachiteration of the iterative algorithm may be adjusted, where thegenerated output may be the final motion vector when the iterativealgorithm is finished with its iterations. Power may also be dynamicallyturned off to the circuitry that makes motion estimation, if said motionestimation is not needed, such as, for example, duringmotion-compensated frame rate up-conversion.

FIG. 1 is an exemplary diagram of a portion of a mobile terminal, whichmay be utilized in connection with an embodiment of the invention.Referring to FIG. 1, there is shown a mobile terminal 100. The mobileterminal 100 may comprise an image sensor 110, an image processor 112, aprocessor 114, and a memory block 116. The image sensor 110 may comprisesuitable circuitry and/or logic that may enable capture of lightintensity at a plurality of colors, such as, for example, red, green,and blue. The captured light intensity levels may be further processedas video and/or still photograph outputs. These color levels may beconverted, for example, to the YUV color space and the resulting colorinformation may be communicated to, for example, the image processor 112for further processing.

The image processor 112 may comprise suitable circuitry and/or logicthat may enable processing of video information. The processing maycomprise, for example, compressing the video information from the imagesensor 110. The processing may also comprise, for example, decompressingvideo information communicated to the mobile terminal 100. Theprocessing may comprise, for example, power-aware motion estimationvideo processing. The processor 114 may determine the mode of operationof various portions of the mobile terminal 100. For example, theprocessor 114 may set up data registers in the image processor block 112to allow direct memory access (DMA) transfers of video data to thememory block 116. The processor 114 may also communicate instructions tothe image sensor 110 to initiate capturing of images. The memory block116 may be used to store image data that may be processed andcommunicated by the image processor 112. The memory block 116 may alsobe used for storing code and/or data that may be used by the processor114. The memory block 116 may also be used to store data for otherfunctionalities of the mobile terminal 100. For example, the memoryblock 114 may store data corresponding to voice communication.

In operation, the processor 114 may initiate image capture by the imagesensor 110. The image sensor 110 may communicate the video datacorresponding to the captured images to the image processor 112. Theimage processor 112 may, for example, compress the video data in asuitable format for storing as a file. The image processor 112functionalities that enable power-aware motion detection are discussedwith respect to FIGS. 2-5. The video data in the memory block 116 may befurther processed by, for example, the processor 114.

FIGS. 2A and 2B are exemplary image lattices illustrating motion vectorestimations, in accordance with an embodiment of the invention.Referring to FIG. 2A, there is shown the image lattices 200, where theimage lattice 200 may be a representation of a previous frame F^(K-1).The image lattice 200 may comprise a plurality of blocks, where eachblock may comprise, for example, M pixels in a horizontal direction andN pixels in a vertical direction. For example, M and N may each be 16.Referring to FIG. 2B, there is shown the image lattice 210, where theimage lattice 210 may be a representation of a present frame F^(K). Theimage lattice 210 may be similar to the image lattice 200 with respectto the number of blocks in the image and where each block comprises M×Npixels. The image processor 112 may, for example, process image latticesto determine motion estimation between various frames.

Motion estimation may be an effective but high-complexity technique forvideo compression and other processing systems. To estimate the motionof a block of video, for example, the block 211 in the frame F^(K), asearch may be initiated to locate a most similar block, or a bestmatched block, in a previous frame. Based on the search, a motion vectormay be generated that may indicate motion from a block in the previousframe to a block in the present frame. For example, the best match forthe block 211 in the present frame F^(K) may be the block 202 in theprevious frame F^(K-1). Accordingly, the motion vector for the block 211may indicate movement from the block 202 in the frame F^(K-1) to theblock 211 in the present frame F^(K). To search for the best matchedblock, a Full Search (FS) motion estimation may be used, where thealgorithm may search through all possible candidates in a search range.This may result in smallest error in matching, or a smallest cost value,but the FS motion estimation may use most resources and most searchtime.

Other search methods, which may have a higher cost value, but a moreefficient use of resources, and a shorter search time may comprise, forexample, a multi-step search, a hierarchical search, and a diamondsearch. Basically, efficiency may be gained by skipping some candidateblocks so that the computational complexity may be lower but the searchresult may still be close to the best-matched result. Other methods ofsearching may comprise, for example, a pixel truncation scheme, wheresome lower bits of a coefficient may be truncated during motionestimation search, and a sub-sampling scheme, which may sub-sample theimage to a smaller image for motion estimation search. Accordingly,these various methods may be combined to reduce a search time whilelimiting loss of precision. A power-aware motion estimation may use, forexample, various searching methods with variable search range, andpre-constrained searching candidate numbers, for example, where thelimitations may increase for low power instances.

The visual motion estimation may comprise an optimization problem thatsearches the optimal optical flow estimate given the visual observation,the matching and flow model, and, possibly, additional input from highercognitive levels. These inputs and models may impose individualconstraints on the solution. Thus, finding an optimal solution may becalled a constraint optimization problem, where its mathematicalformulation and the computational approach chosen may depend on theconstraints imposed.

The first constraint chosen may be the minimum displacement constraint.This may come various motion estimation methods that apply the blockmatching explicitly. The displacement per time unit in image space mayserve as a measure for image motion. Typically, the matching method maybe described by the following least-squares error equation:

E(x,y,t;v _(x) ,v _(y) ,Δt)=(P(x,y,t)−P(x+v _(x) ,y+v _(y),t+Δt))²,  [1]

where P may describe the brightness distribution in the image space andtime on a Cartesian coordinate system, and v_(x) and v_(y) may betranslational displacements, also called a motion vector, in imagespace. The coordinates v_(x) ^(t) and v_(y) ^(t) may be determined for atime t for a given Δt such that E(x,y,t;v_(x) ^(t),v_(y) ^(t),Δt) isminimized. Accordingly, the equation [1] may be expressed as:

{right arrow over (v)} ^(t)=(v _(x) ^(t) ,v _(y) ^(t))=arg min_(v) _(x)_(,v) _(y) E(x,y,t;v _(x) ,v _(y) ,Δt).  [12]

The instantaneous estimate of optical flow in an image space may be itsvector field, which may be expressed as:

$\begin{matrix}{{{\overset{\rightharpoonup}{v}}_{t}\left( {x,y,t} \right)} = {\left( {\frac{v_{x}^{t}}{t},\frac{v_{y}^{t}}{t}} \right).}} & \lbrack 3\rbrack\end{matrix}$

Using Taylor expansion, P(x,y,t) may be expressed as:

$\begin{matrix}{{P\left( {{x + v_{x}},{y + v_{y}},{t + {\Delta \; t}}} \right)} = {{P\left( {x,y,t} \right)} + {\frac{\partial P}{\partial x}v_{x}} + {\frac{\partial P}{\partial y}v_{y}} + {\frac{\partial P}{\partial t}\Delta \; t} + {ɛ\left( o^{2} \right)}}} & \lbrack 4\rbrack\end{matrix}$

where ε(o²) may comprise higher order terms that are neglected. Thus,equation [1] may be expressed as:

$\begin{matrix}{{E\left( {x,y,{t;v_{x}},v_{y},{\Delta \; t}} \right)} = {\left( {{\frac{\partial P}{\partial x}v_{x}} + {\frac{\partial P}{\partial y}v_{y}} + {\frac{\partial P}{\partial t}\Delta \; t}} \right)^{2}.}} & \lbrack 5\rbrack\end{matrix}$

An exemplary visual motion estimation may be determined utilizing a flowmodel that imposes assumptions related to properties of the expectedoptical flow. For example, an assumption may be that the optical flowestimate is smooth. The smoothness constraint S({right arrow over (v)})may be defined as a quadratic measure of how much the flow field variesacross the image space:

$\begin{matrix}{{S\left( \overset{\rightharpoonup}{v} \right)} = {\left( \frac{\partial v_{x}}{\partial x} \right)^{2} + \left( \frac{\partial v_{x}}{\partial y} \right)^{2} + \left( \frac{\partial v_{y}}{\partial x} \right)^{2} + \left( \frac{\partial v_{y}}{\partial y} \right)^{2} + \left( \frac{\partial v_{x}}{\partial t} \right)^{2} + \left( \frac{\partial v_{y}}{\partial t} \right)^{2}}} & \lbrack 6\rbrack\end{matrix}$

Smoothness may assume, for example, that locations in images close toeach other may be more likely to belong to a same moving region, andthus may be more likely to have a same visual motion. Smoothness may beclosely related to a motion field of rigid objects. However, thesmoothness constraints may be violated at locations where motiondiscontinuities are likely to occur, such as, for example, moving objectboundaries. This may be due to judder, or temporal artifacts that may beassociated with moving images that have been sampled at one rate andconverted for display at another rate.

Another assumption of the flow model may be, for example, that someoptical flow vectors may be derived from a known reference motionvector. This may be referred to, for example, as a bias constraint. Suchconstraint may be defined as:

B({right arrow over (v)},v _(ref))=(v _(x) −v _(x) _(ref) )²+(v _(y) −v_(y) _(ref) )²,  [7]

where B may measure how much the estimated optical flow vector maydeviate from a given reference motion {right arrow over(v)}_(ref)=(v_(x) _(ref) ,v_(y) _(ref) ). The bias constraint may permitthe visual motion estimation algorithm to incorporate and apply a prioriinformation about expected motion. For example, the given referencemotion may be a global motion vector. The reference motion vector {rightarrow over (v)}_(ref)=(v_(x) _(ref) ,v_(y) _(ref) ) may specify a goodoptical flow estimate under conditions where the visual input may beambiguous and unreliable. The reference motion vector {right arrow over(v)}_(ref)=(v_(x) _(ref) ,v_(y) _(ref) ) may be used to adapt the visualmotion estimation algorithm to different visual conditions. The biasconstraint may incorporate additional information that may allow abetter estimate of visual motion.

The above constraints may be combined such that for a continuous inputof the brightness distribution P(x,y,t) on an image region Ω⊂R², theproblem to solve may be to find v_(x) and v_(y) for a given timeduration Δt such that the cost function

$\begin{matrix}{{J\left( {\overset{\rightharpoonup}{v},w_{S},w_{B}} \right)} = {\int_{\Omega}{\left( {E + {w_{S} \cdot S} + {w_{B} \cdot B}} \right){\Omega}}}} & \lbrack 8\rbrack\end{matrix}$

may be minimal, with {right arrow over (v)}″={right arrow over (0)}along ∂Ω and w_(S)>0, and w_(B)≧0. The relative weights of theconstraints may be set by the parameters w_(S) and w_(B). When thevisual motion estimation algorithm is first initialized for use, theseparameters may be assumed to be constant. However, since theseparameters may be considered as a function of time, location, and thecurrent motion estimation, these parameters may be variable. Weightingparameters and the reference motion vector, which may be designdependent, may allow the visual motion estimation algorithm to accountfor different models of visual motion perception. The motion estimationformulated as the constraint optimization problem expressed by theequation [8] may have a unique solution that may depend continuously oninput data.

Although equation [8] may be formulated on a continuous image spaceΩ⊂R², video data may be provided by an imaging device as discretetemporal pictures with discrete spatial resolution. For example, if adiscrete image space is defined on an orthogonal lattice, where eachnode is labeled by two integer values iε[0, 1, . . . , H−1] and jε[0, 1,. . . , V−1], equation [8] may be reduced to a finite number oflocations. To model the cost function for implementation of equation[8], the (i,j) lattice may be partitioned to p×q blocks, where eachblock may comprise m×n pixels. Accordingly, iε[pm, . . . , pm+m−1] where

${p \in \left\lbrack {0,1,\ldots \;,{\left\lfloor \frac{H}{m} \right\rfloor - 1}} \right\rbrack},$

and jε[qn, . . . , qn+n−1] where

$q \in {\left\lbrack {0,1,\ldots \;,{\left\lfloor \frac{V}{n} \right\rfloor - 1}} \right\rbrack.}$

Since a single motion vector may be determined for each block, there maybe p×q motion vectors in an image space, such as, for example, the imagelattice 200 and the image lattice 210. Thus, the estimation of theoptical flow motion vector may be modeled as a causal discreteconstraint optimization process with the cost function for the block (p,q) expressed as:

$\begin{matrix}{{J\left( {\overset{\rightharpoonup}{v},w_{S},w_{B}} \right)} = {{\sum\limits_{i = {pm}}^{{pm} + m - 1}{\sum\limits_{j = {qn}}^{{qn} + n - 1}\left( {{P\left( {x_{i},y_{j},k} \right)} - {P\left( {{x_{i} + v_{x}},{y_{j} + v_{y}},{k - 1}} \right)}} \right)^{2}}} + {w_{S} \cdot \left( {\left( {v_{x} - v_{x,{({{p - 1},q})}}^{k}} \right)^{2} + \left( {v_{y} - v_{y,{({{p - 1},q})}}^{k}} \right)^{2} + \left( {v_{x} - v_{x,{({p,{q - 1}})}}^{k}} \right)^{2} + \left( {v_{y} - v_{y,{({p,{q - 1}})}}^{k}} \right)^{2} + \left( {v_{x} - v_{x,p,q}^{k - 1}} \right)^{2} + \left( {v_{y} - v_{y,{({p,q})}}^{k - 1}} \right)^{2}} \right)} + {w_{B} \cdot \left( {\left( {v_{x} - v_{x,{ref}}} \right)^{2} + \left( {v_{y} - v_{y_{ref}}} \right)^{2}} \right)}}} & \lbrack 9\rbrack\end{matrix}$

where the partial derivatives of the flow vectors in the smoothness termin equation [8] may be replaced by a difference operator, and an index‘k’ may indicate the frame number. The term v_(b,(c,d)) ^(a) mayindicate a previously estimated motion vector in the frame ‘a’ for the‘b’ direction and at the grid location (c, d). While an embodiment ofthe invention may use the equation [9], other variations of the equation[9] may be used. For example, the smoothness term with v_(b,(c,d)) ^(a)may be altered.

An embodiment of the invention may use a block matchingmotion-estimation algorithm to minimize a cost function for equation[9]. The equation [9] may have a unique solution as in the continuouscase. Accordingly, it follows that

J′({right arrow over (v)},w _(S) ,w _(B))=0  [10]

may be a sufficient condition for the unique solution. J({right arrowover (v)},w_(S),w_(B)) may be assumed to be sufficiently regular and atleast twice continuously differentiable. Thus, gradient descent on thecost function [9] may be used as an update strategy for an iterativemotion estimation algorithm. The changes of each motion vector componentv_(x) and v_(y) may be negatively proportional to the partial gradientsof the cost function, and may be expressed as:

$\begin{matrix}{{{\overset{.}{v}}_{x} \propto {{- \frac{\partial{J\left( {\overset{\rightharpoonup}{v},w_{S},w_{B}} \right)}}{\partial v_{x}}}\mspace{14mu} {and}\mspace{14mu} {\overset{.}{v}}_{y}} \propto {- \frac{\partial{J\left( {\overset{\rightharpoonup}{v},w_{S},w_{B}} \right)}}{\partial v_{y}}}}\mspace{14mu}} & \lbrack 11\rbrack\end{matrix}$

until steady state may be reached. This may lead to the following linearpartial differential equations, which may be jointly referred to asequation [12]:

$\begin{matrix}{{{\overset{.}{v}}_{x} = {{- A} \cdot \left( {{\sum\limits_{i = {pm}}^{{pm} + m - 1}{\sum\limits_{j = {qn}}^{{qn} + n - 1}{B_{x_{i},y_{j}} \cdot {E\left( {x_{i},y_{j},{k;v_{x}},v_{y},1} \right)}}}} + {w_{S} \cdot \left( {{3v_{x}} - v_{x,{({{p - 1},q})}}^{k} - v_{x,{({p,{q - 1}})}}^{k} - v_{x,{({p,q})}}^{k - 1}} \right)} + {w_{B} \cdot \left( {v_{x} - v_{x_{ref}}} \right)}} \right)}}{{\overset{.}{v}}_{y} = {{- A} \cdot \left( {{\sum\limits_{i = {pm}}^{{pm} + m - 1}{\sum\limits_{j = {qn}}^{{qn} + n - 1}{C_{x_{i},y_{j}} \cdot {E\left( {x_{i},y_{j},{k;v_{x}},v_{y},1} \right)}}}} + {w_{S} \cdot \left( {{3v_{y}} - v_{y,{({{p - 1},q})}}^{k} - v_{y,{({p,{q - 1}})}}^{k} - v_{y,{({p,q})}}^{k - 1}} \right)} + {w_{B} \cdot \left( {v_{y} - v_{y_{ref}}} \right)}} \right)}}} & \lbrack 12\rbrack\end{matrix}$

whereE(x_(i),y_(j),k;v_(x),v_(y),1)=E(x_(i),y_(j),k)−E(x_(i)+v_(x),y_(j)+v_(y),k−1),‘A’ is a positive constant, and

${B_{x_{i},y_{j}} = \frac{\partial P}{\partial x_{i}}},{C_{x_{i},y_{j}} = \frac{\partial P}{\partial y_{j}}},$

as derived from equation [4].

The Steepest-Descent (SD) method may be used for finding a solution tothe equation [12]. The SD method may take the previous result {rightarrow over (v)}(ρ−1) and update it using the instantaneous gradient ofthe temporal squared error, for example, E(x_(i),y_(j)k;v_(x),v_(y),1),to get the following recursive equation:

{right arrow over (v)}(ρ)={right arrow over (v)}(ρ−1)+A·({right arrowover (f)}({right arrow over (v)}(0),{right arrow over (v)}_(p-1,q)) ^(k),{right arrow over (v)} _((p,q-1)) ^(k) ,{right arrow over (v)} _((p,q))^(k-1) ,{right arrow over (v)} _(ref))−(3w _(S) +w _(B))·{right arrowover (v)}(ρ−1)),  [13]

where

${\overset{\rightharpoonup}{v}(\rho)} = \begin{pmatrix}{v_{x}(\rho)} \\{v_{y}(\rho)}\end{pmatrix}$

and ρ is the iteration step. To simplify the operation, B_(x) _(i) _(,y)_(j) and C_(x) _(i) _(,y) _(j) may be assumed to be constants for eachm×n block. Thus,

$\begin{matrix}{{\overset{\rightharpoonup}{f}\left( {{\overset{\rightharpoonup}{v}(0)},{\overset{\rightharpoonup}{v}}_{({{p - 1},q})}^{k},{\overset{\rightharpoonup}{v}}_{({p,{q - 1}})}^{k},{\overset{\rightharpoonup}{v}}_{({p,q})}^{k - 1},{\overset{\rightharpoonup}{v}}_{ref}} \right)} = \begin{pmatrix}{{{- B_{p,q}}{\sum\limits_{i = {pm}}^{{pm} + m - 1}{\sum\limits_{j = {qn}}^{{qn} + n - 1}{E\begin{pmatrix}{x_{i},y_{j},{k;}} \\{{{\overset{\rightharpoonup}{v}}^{T}(0)},1}\end{pmatrix}}}}} + {w_{S} \cdot \begin{pmatrix}\begin{matrix}{v_{x,{({{p - 1},q})}}^{k} +} \\{v_{x,{({p,{q - 1}})}}^{k} +}\end{matrix} \\v_{x,{({p,q})}}^{k - 1}\end{pmatrix}} + {w_{B} \cdot v_{x_{ref}}}} \\{{{- C_{p,q}}{\sum\limits_{i = {pm}}^{{pm} + m - 1}{\sum\limits_{j = {qn}}^{{qn} + n - 1}{E\begin{pmatrix}{x_{i},y_{j},{k;}} \\{{{\overset{\rightharpoonup}{v}}^{T}(0)},1}\end{pmatrix}}}}} + {w_{S} \cdot \begin{pmatrix}\begin{matrix}{v_{y,{({{p - 1},q})}}^{k} +} \\{v_{y,{({p,{q - 1}})}}^{k} +}\end{matrix} \\v_{y,{({p,q})}}^{k - 1}\end{pmatrix}} + {w_{B} \cdot v_{y_{ref}}}}\end{pmatrix}} & \lbrack 14\rbrack\end{matrix}$

The initial motion vector {right arrow over (v)}(0) may be computed by alower-resolution block matching search algorithm and B_(x) _(p) _(,y)_(q) and C_(x) _(p) _(,y) _(q) may be estimated, for example, by theblock boundary pixel difference. Equation [13] may be, for example, oneiteration of the steepest decent algorithm. The parameter A, which maybe referred to as the step-size, may control the size of the incrementalcorrection applied to the motion vector from one iteration to the next.The parameter A may need to be determined such that the relationshipdescribed by equation [13] is stable, or converges to a solution.

For frames that comprise fast moving scenes, the term {right arrow over(v)}_((p,q)) ^(k-1) in equations [13] and [14] may be replaced by amotion vector of the block in the k−1 frame pointed by the vector)−{right arrow over (v)}(0) from the block (p,q) in the frame k. Theminus sign may indicate that the vector may be from a previous frame.Accordingly, a motion vector for a block in a present frame may be setto be equal to the motion vector for a block in a previous frame. Forexample, if the motion vector for the block 202 points to the block 211,the block 211 may be given a motion vector that may be the same as themotion vector for the block 201. A data flow diagram for animplementation using equation [13] is disclosed with respect to FIG. 3.

FIG. 3 is an exemplary block diagram illustrating data flow forpower-aware motion estimation, in accordance with an embodiment of theinvention. Referring to FIG. 3, there is shown a portion of the mobileterminal 100 that comprise an adaptive power control block 300, anadaptive motion estimation engine 310, and memory 320. The adaptivepower control block 300 may comprise suitable logic, circuitry, and/orcode that may enable controlling power to the adaptive motion estimationengine 310. The power control functionality may be, for example,programmable based on whether the mobile terminal 100 may be in alow-power state. In instances when the mobile terminal 100 is in alow-power state, power to the adaptive motion estimation engine 310 maybe turned off by the adaptive power control block 300. In instances whenthe mobile terminal 100 is in a full-power state, power to the adaptivemotion estimation engine 310 may be turned on by the adaptive powercontrol block 300. The threshold voltage(s) at which the adaptive powercontrol block 300 may turn on or off power to the adaptive motionestimation engine 310 may be programmed by, for example, the processor114. The processor 114 may also communicate a command to the adaptivepower control block 300 to turn off the adaptive motion estimationengine 310. The processor 114 may communicate a power control commandbased on, for example, a user's selection of a power model for use bythe mobile terminal 100.

The number of iterations for motion estimation may also be based on, forexample, limitation of the power source and/or the complexity of videodata. For example, while the mobile terminal 100 may be in a full-powerstate, if video data complexity is deemed to be decreasing, then thenumber of iterations for motion estimation may be reduced. If the videodata complexity is simple enough, the motion estimation engine may beturned off. The determination of video data complexity may be designdependent. Additionally, even with complex video data, in instanceswhere voltage level of, for example, a battery (not shown) isdecreasing, the number of iterations for motion estimation may bereduced. Video data complexity may be determined by, for example, theimage processor 112 and/or the processor 114. Accordingly, the imageprocessor 112 and/or the processor 114 may program the number ofiterations to execute for motion estimation for various levels of videodata complexity.

The adaptive motion estimation engine 310 may be an implementation of,for example, the equation [13]. The adaptive motion estimation engine310 may comprise a low resolution search block 312, an initial motionvector generator block 314, amplitude adjuster blocks 316, 324, 326, and328, a summer block 318, an iteration control block 320, and a delayblock 322. The low resolution search block 312 may comprise suitablelogic, circuitry, and/or code that may sample video blocks in a videoframe to generate lower resolution video blocks for the video frame. Thelow resolution search block 312 may then determine a global motionvector for a frame. The low resolution search block 312 may also searchfor best matches for each low resolution video block to generate lowresolution motion vectors for each video block in the video frame, wherethe scope of search may be controlled by, for example, the imageprocessor 112 and/or the processor 114. Accordingly, the scope of thesearch may be the blocks in the video frame, or a portion of the blocksin the video frame.

The initial motion vector generator block 314 may comprise suitablelogic, circuitry, and/or code that may receive input data from the lowresolution search block 312 as well as motion vectors from a previousvideo frame. Various embodiments of the invention may use differentmotion vectors from a previous frame, depending on the solution used forthe equation [13].

The amplitude adjuster blocks 316, 324, 326, and 328 may comprisesuitable logic, circuitry, and/or code that may enable multiplying aninput digital data by a suitable value, where the value used may becontrolled by, for example, the image processor 112 and/or the processor114.

The summer block 318 may comprise suitable logic, circuitry, and/or codethat may enable adding digital data from a plurality of sources tooutput a single digital data. The iteration control block 320 maycomprise suitable logic, circuitry, and/or code that may enablecontrolling a number of iterations used in determining a motion vectorfor a block in a video frame. The delay block 322 may comprise suitablelogic, circuitry, and/or code that may enable providing suitable delaysuch that a motion vector for a block may be processed to generate anext iteration of a motion vector.

Memory 320 may be, for example, a part of the memory block 116, and maystore the motion vectors generated by the adaptive motion estimationengine 310. The motion vectors stored in the memory 320 may becommunicated to, for example, the low resolution search block 312 and/orthe initial motion vector generator block 314.

In operation, video data for a video block in a present frame F^(K) maybe communicated to the scene adaptive auto shut-off block 300. The sceneadaptive auto shut-off block 300 may communicate the video data to thelow resolution search block 312 if it determines that the adaptivemotion estimation engine 310 may be turned on to generate motionvectors. If the scene adaptive auto shut-off block 300 determines thatthe adaptive motion estimation engine 310 may need to be shut off, thevideo data may be, for example, further processed by the image processor112 without motion estimation.

The low resolution search block 312 may sample the video data togenerate a lower resolution video block that may correspond to the videodata from the scene adaptive auto shut-off block 300. The scene adaptiveauto shut-off block 300 may also be controlled via a shut-off commandfrom, for example, the processor 114. For example, the processor 114 maydynamically communicate a command to the scene adaptive auto shut-offblock 300 to shut off power to the adaptive motion estimation engine 310when motion estimation is not needed. For example, when the mobileterminal 100 is performing motion-compensated frame rate up-conversion,a frame may be repeated and motion estimation may not be needed.Accordingly, the processor 114 may communicate a command to have theadaptive motion estimation engine 310 turned off to save power.

The low resolution search block 312 may then generate a low resolutionmotion vector {right arrow over (v)}(0) for each lower resolution block.The low resolution search block 312 may also output a global motionvector {right arrow over (v)}_(ref), which may be, for example, storedin the low resolution search block 312 by the processor 114. The motionvectors may be communicated to the vector generator block 314. Thevector generator block 314 may use the motion vectors from the lowresolution search block 312 and motion vectors for blocks from aprevious frame to determine a motion vector for a block in the presentframe. The motion vectors for blocks from a previous frame may bestored, for example, in the memory 320. For example, a motion vector{right arrow over (f)} for the block 212, which may also be referred toas block (p,q) in the present frame F^(K), may be determined based onmotion vectors for the blocks 201, 202, and 203 in the previous frameF^(K-1), as well as the global motion vector {right arrow over(v)}_(ref) for the present frame F^(K) and the low resolution motionvector {right arrow over (v)}(0) for the block 212 in the present frameF^(K). The motion vector {right arrow over (f)} may be represented bythe equation [14].

The motion vector {right arrow over (f)} may be communicated to theamplitude adjuster block 316, which may multiply the motion vector{right arrow over (f)} by a parameter A, where the parameter may be aprogrammable parameter. The parameter A may be, for example, a positivevalue that may be chosen to ensure convergence to a solution. The outputof the amplitude adjuster block 316 may be constant until the motionvector is generated at the final iteration. The output of the amplitudeadjuster block 316 may be communicated to the summer block 318, whichmay combine the outputs of the amplitude adjuster block 316, 324, and328. The output of the summer block 318 may be referred to as a presentiteration motion vector {right arrow over (v)}(ρ). The iteration controlblock 320 may determine whether a number of iterations selected formotion estimation may have occurred. If so, the present iteration motionvector {right arrow over (v)}(ρ) may be saved as a final motion vectorfor a block, for example, the block (p,q), in the memory 320.Accordingly, the motion vectors for all blocks in a frame may be savedin the memory 320 for future use, including for generation of motionvectors for a subsequent frame.

If more iterations are to take place to generate the motion vector, thepresent iteration motion vector {right arrow over (v)}(ρ) may becommunicated to the delay block 322. The delay block 322 may delay thepresent iteration motion vector {right arrow over (v)}(ρ) such that itmay be used as a previous iteration motion vector {right arrow over(v)}(ρ−1), as shown in the equation [14]. Various embodiments of theinvention may, for example, set the output of the delay block 322 tozero at the start of a first iteration, or to a known pre-determinedvalue.

The previous iteration motion vector {right arrow over (v)}(ρ−1) may becommunicated to the amplitude adjuster blocks 324 and 326. The amplitudeadjuster blocks 324 may be output a motion vector that may be identicalto the motion vector {right arrow over (v)}(ρ−1). The output of theamplitude adjuster block 324 may be communicated to the summer block 318for generating a subsequent present iteration motion vector {right arrowover (v)}(ρ). The amplitude adjuster block 326 may, for example,multiply the motion vector {right arrow over (v)}(ρ−1) by a parameter of“3Ws+Wb,” where the weighting parameter Ws may be greater than zero andthe weighting parameter Wb may be greater than or equal to zero. Variousembodiments of the invention may initially set the weighting parametersWs and Wb to some constant values, where the constant values may dependon, for example, different models of visual motion perception.Accordingly, the initial values of the weighting parameters Ws and Wbmay be design and/or implementation dependent.

The output of the amplitude adjuster block 326 may be communicated tothe amplitude adjuster block 328, which may multiply the datacommunicated from the amplitude adjuster block 326 by a parameter of“−A.” The parameter −A may be a negative of the parameter A used by theamplitude adjuster block 326 316. The output of the amplitude adjusterblock 328 may be communicated to the summer block 316. The output of theamplitude adjuster blocks 324 and 328 may be referred to as a feedbackmotion vector.

FIG. 4 is an exemplary graph illustrating power consumption versusnumber of iterations in determining motion estimation, in accordancewith an embodiment of the invention. Referring to FIG. 4, there is shownthe graph 400 that may represent, for example, power used in processingvideo. For example, graph point 402 may indicate zero iterations forgenerating a motion vector. The number of iterations may be set to zero,for example, when the adaptive motion estimation engine 310 may beturned off. Other embodiments of the invention may allow a separatestate where the adaptive motion estimation engine 310 may be turned on,but with zero iterations, and where the adaptive motion estimationengine 310 may be turned off. Accordingly, video processing may resultin increased power usage as the number of iterations for motion vectorestimation increases, until the maximum number of iterations is reached.

Various embodiments of the invention may, therefore, control powerconsumption for video processing by turning on or off, for example, theadaptive motion estimation engine 310. Power consumption may also bechanged by adjusting the number of iterations allowed for determining amotion vector for a block of video in a present frame. Accordingly, bysuitably determining power consumption versus operation of the adaptivemotion estimation engine 310, various power models may be generated. Auser of the mobile terminal 100, for example, may then select a desiredpower model for video processing, and the image processor 114 may beappropriately configured.

For example, the processor 114 may communicate a command to the adaptivepower control block 300 to turn off power to the adaptive motionestimation engine 310. The processor 114 may also communicate a commandto the power control block 300 to turn on power to the adaptive motionestimation engine 310. When the adaptive motion estimation engine 310 isturned on, various parameters may be adjusted, for example, by theprocessor 114. For example, the parameters A, Ws, Wb, and −A for theamplitude adjuster blocks 316, 324, 326, and 328, respectively may beset to desired values. Additionally, the number of iterations may be setfor the iteration control block 320.

The scene adaptive auto shut-off block 300 may also be controlled via ashut-off command from, for example, the processor 114 to save power forcertain applications, such as, for example, motion-compensated framerate up-conversion where a frame may be repeated to save power.

In accordance with an embodiment of the invention, aspects of anexemplary system may comprise the scenic adaptive auto shut-off block300, the adaptive motion estimation engine 310, and memory 320. Thescenic adaptive auto shut-off block 300 may control, for example, powerto the adaptive motion estimation engine 310. The scenic adaptive autoshut-off block 300 may either turn on or turn off power to the adaptivemotion estimation engine 310 based on video content of video datacommunicated to the scenic adaptive auto shut-off block 300 and/oravailable power for the mobile terminal 100. The power level at whichthe scenic adaptive auto shut-off block 300 turns on or off the adaptivemotion estimation engine 310 may be design and/or implementationdependent. For example, the threshold value for the power level may beset by the processor 114 to a default value, or according to a powermodel selected by a user.

When the power to the adaptive motion estimation engine 310 is turnedon, the adaptive motion estimation engine 310 may make motionestimations for video data. The motion estimations may comprise a motionvector for each block of video data in a frame. For example, the lowresolution search block 312 may generate a low resolution block fromvideo data for the block 212 in the present frame F^(K) by sampling thevideo data in the block 212. The low resolution search block 312 maythen perform block matching for the low resolution block of video datawith one or more similarly sampled low resolution blocks of video datain a previous frame, for example, the frame F^(K-1). The block matchingmay result in a low resolution motion vector for the block 212 in thepresent frame. The low resolution search block 312 may also output aglobal motion vector.

The initial motion vector generator block 314 may generate an initialmotion vector, or a preliminary motion vector, based on the lowresolution motion vector for the block 212 and the global motion vector.The initial motion vector may also be based on, for example, motionvectors from the previous frame. The initial motion vector may be summedby the summer block 318 with a feedback motion vector, where thefeedback motion vector may be iteratively adjusted. The feedback motionvector may be generated by processing the output of the summer block 318via the delay block 322, and the amplitude adjuster blocks 324, 326, and328.

The number of iterations may be controlled, for example, by theprocessor 114. The number of iterations used for the iterative algorithmin generating the final motion vector may be, for example, communicatedby the processor 114 to the iteration control block 320. After thenumber of iterations allowed by the iteration control block 320, thesummed result of the initial motion vector and the feedback motionvector may be saved in memory 320 as a final motion vector for the block212. The final motion vector for each of the blocks in the present frameF^(K) may be used to generate motion vectors for blocks of video data ofa subsequent frame.

Various embodiments of the invention may also adjust the number ofiterations based on video content of video data communicated to thescenic adaptive auto shut-off block 300 and/or levels of available powerfor the mobile terminal 100. The power levels at which the scenicadaptive auto shut-off block 300 may adjust the number of iterations maybe design and/or implementation dependent. For example, the thresholdvalues for the power levels may be set by the processor 114 to a defaultvalue, or according to a power model selected by a user. The processor114, for example, may also control the incremental correction for anoutput generated at each iteration of the iterative algorithm, whereinthe generated output may be the final motion vector when the iterativealgorithm is finished with its iterations. A processor, such as, forexample, the processor 114 may also communicate a command to the scenicadaptive auto shut-off block 300 to turn off power to the adaptivemotion estimation engine 310. This may occur when motion estimation isnot needed during such times as, for example, when motion-compensatedframe rate up-conversion repeats a frame.

Another embodiment of the invention may provide a machine-readablestorage, having stored thereon, a computer program having at least onecode section executable by a machine, thereby causing the machine toperform the steps as described herein for power-aware motion estimationvideo processing.

Accordingly, the present invention may be realized in hardware,software, or a combination of hardware and software. The presentinvention may be realized in a centralized fashion in at least onecomputer system, or in a distributed fashion where different elementsare spread across several interconnected computer systems. Any kind ofcomputer system or other apparatus adapted for carrying out the methodsdescribed herein is suited. A typical combination of hardware andsoftware may be a general-purpose computer system with a computerprogram that, when being loaded and executed, controls the computersystem such that it carries out the methods described herein.

The present invention may also be embedded in a computer programproduct, which comprises all the features enabling the implementation ofthe methods described herein, and which when loaded in a computer systemis able to carry out these methods. Computer program in the presentcontext means any expression, in any language, code or notation, of aset of instructions intended to cause a system having an informationprocessing capability to perform a particular function either directlyor after either or both of the following: a) conversion to anotherlanguage, code or notation; b) reproduction in a different materialform.

While the present invention has been described with reference to certainembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted withoutdeparting from the scope of the present invention. In addition, manymodifications may be made to adapt a particular situation or material tothe teachings of the present invention without departing from its scope.Therefore, it is intended that the present invention not be limited tothe particular embodiment disclosed, but that the present invention willcomprise all embodiments falling within the scope of the appendedclaims.

1. A method, comprising: generating a motion vector for estimatingmotion in video data by block matching a first block of video data in apresent frame with at least one block of video data in a previous frame;applying an incremental correction to the motion vector; and dynamicallyadjusting a number of times the applying the incremental correction tothe motion vector is performed for the motion vector based at least inpart on a selected one of a plurality of power models, the power modelscorresponding to varying power consumption requirements.
 2. The methodof claim 1, further comprising dynamically adjusting the number of timesthe applying the incremental correction to the motion vector isperformed for the motion vector based at least in part on availablepower.
 3. The method of claim 1, further comprising generating eachincremental correction based at least in part on a respective motionvector that corresponds to one block of video data from the previousframe.
 4. The method of claim 1, further comprising dynamicallydisabling motion estimation using the motion vector that has beenincrementally corrected based at least in part on available power. 5.The method of claim 1, further comprising dynamically enabling motionestimation using the motion vector that has been incrementally correctedbased at least in part on available power.
 6. The method of claim 1,further comprising storing the motion vector that has been incrementallycorrected in memory for use in estimating motion for a subsequent frame.7. The method of claim 1, further comprising: determining whether motionestimation is needed for the video data; and dynamically powering offcircuitry employed to perform the motion estimation in response todetermining that the motion estimation is not needed for the video data.8. The method of claim 1, further comprising: sampling the first blockof video data to generate a lower resolution block of video data; andwherein the lower resolution block of video data is used for the blockmatching with similarly sampled lower resolution blocks of video datafor the previous frame.
 9. A system, comprising: circuitry configuredto: generate a motion vector for estimating motion in video data byblock matching a first block of video data in a present frame with atleast one block of video data in a previous frame; apply at least oneincremental correction to the motion vector; and dynamically adjust anumber of times the motion vector is incrementally corrected based atleast in part on a measure of power available to the circuitry.
 10. Thesystem of claim 9, wherein the circuitry is further configured to applythe at least one incremental correction according to an iterativealgorithm.
 11. The system of claim 9, wherein the circuitry is furtherconfigured to dynamically adjust the number of times the motion vectoris incrementally corrected based at least in part on a selected one of aplurality of power models, the power models corresponding to varyingpower consumption requirements.
 12. The system of claim 9, wherein thecircuitry is further configured to generate each one of the at least oneincremental correction based at least in part on a respective motionvector that corresponds to one block of video data from the previousframe.
 13. The system of claim 9, wherein the circuitry is furtherconfigured to dynamically disable motion estimation using the motionvector that has been incrementally corrected based at least in part onthe measure of power available to the circuitry.
 14. The system of claim9, wherein the circuitry is further configured to dynamically enablemotion estimation using the motion vector that has been incrementallycorrected based at least in part on the measure of power available tothe circuitry.
 15. The system of claim 9, wherein the circuitry isfurther configured to store the motion vector that has beenincrementally corrected in memory for use in estimating motion for asubsequent frame.
 16. The system of claim 9, wherein the circuitry isfurther configured to: determine whether motion estimation is needed forthe video data; and dynamically power off a portion of the circuitry inresponse to determining that the motion estimation is not needed for thevideo data.
 17. The system of claim 9, wherein the circuitry is furtherconfigured to: sample the first block of video data to generate a lowerresolution block of video data; and wherein the lower resolution blockof video data is used for the block matching with similarly sampledlower resolution blocks of video data for the previous frame.
 18. Anon-transitory computer-readable medium embodying a program executablein a computing device, comprising: code that generates a motion vectorfor estimating motion in video data by block matching a first block ofvideo data in a present frame with at least one block of video data in aprevious frame; code that iteratively applies at least one incrementalcorrection to the motion vector; and code that dynamically adjusts anumber of iterations employed by the code that iteratively applies theat least one incremental correction to the motion vector based at leastin part on a measure of power available to the computing device.
 19. Thenon-transitory computer-readable medium of claim 18, further comprisingcode that dynamically adjusts the number of iterations employed by thecode that iteratively applies the at least one incremental correction tothe motion vector based at least in part on a selected one of aplurality of power models, the power models corresponding to varyingpower consumption requirements.
 20. The non-transitory computer-readablemedium of claim 18, further comprising: code that samples the firstblock of video data to generate a lower resolution block of video data;and wherein the lower resolution block of video data is used for theblock matching with similarly sampled lower resolution blocks of videodata for the previous frame.